1. Sadayuki Furuhashi
Kumofs
MessagePack
Project
#msgpack @frsyuki
Distributed key-value store
and e

2. About me:
Sadayuki Furuhashi (@frsyki)
Diskless network system
VIVER
kumofs
Distributed key-value store
MessagePack
Cross-language communication library
University of Tsukuba
Master’s cource in Computer Science
supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan
Projects:

3. About me:
Sadayuki Furuhashi
Diskless network system
VIVER
kumofs
Distributed key-value store
MessagePack
Cross-language communication library
University of Tsukuba
Master’s cource in Computer Science
supported by “Mitoh” project run by IPA, Information-technology Promotion Agancy, Japan
Projects:

4. kumofs
kumo = 雲 = cloud

5. What s kumofs?
• Distributed key-value store
• Get, Set, Delete and CAS (Compare-And-Swap)
• Linear scalability as servers added
• both read and write throughput scales-out
• dynamic rebalancing without any impacts on
applications
• Optimized for read-latency
• always zero-hop
• No SPOF

6. http://itpro.nikkeibp.co.jp/article/COLUMN/20090617/331953/
memcachedを超える成果も、Interopで若手技術者がクラウドを支える技術を競う

7. http://gihyo.jp/dev/serial/01/alpha-geek/0041
“小飼弾のアルファギークに いたい♥”

8. Shows and manages hundreds of photos smoothly
> gives lots of accesses on metadata servers
http://yapcasia2009.ficia.com/pl/album/7BF0E66A-A35F-11DE-966B-860A873069EA

9. Why kumofs?
• Need to manage heavy random reads and writes
• Need low latency for user-experience
• Need elastic scalability
• builds small cluster at the first
• enlarges it when number of users increases
• “I won’t wakeup at midnight!”
• Systems that have SPOF rushes administrators to
recover crashed servers even if it’s at midnight.

10. Archtecture
of
kumofs

11. Manager
Gateway
kumo-manager:
Watches kumo-servers to
detect/detach crashed severs
Archtecture of kumofs
kumo-server:
Stores and replicates data
kumo-gateway:
Relays requests from
applications to kumo-servers
Server

12. Application
Server
Manager
Gateway
Manager
Duplicated
(HA)
Server
Server
Server
Server
Server
Application
Gateway
Application
Gateway
Replication
Archtecture of kumofs
Tokyo CabinetTokyo Cabinet

13. hash(key1)
Server A
Server B
Server C
Server D
Consistent Hashing

14. Server A
Server B
Server C
Server D
coordinated by
Server B

15. Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server D
Server A

16. Server A
Server B
Server C
Server D
set
Replicate
Replication mechanism

17. Server A
Server B
Server C
Server D
get
High-availability

18. Server A
Server B
Server C
Server D
get
High-availability

19. Server A
Server B
Server C
Server D
get
High-availability

20. Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server D
Server A
Adding nodes dynamically

21. Server A
Server B
Server C
Server D
coordinated by
Server B
Server C
Server A
Server E
Copies data
Adding nodes dynamically

22. Server A
Server B
Server C
Server D
Server E
Data are moving...
Adding nodes dynamically

23. Server A
Server B
Server C
Server D
Server E
set
get
Adding nodes dynamically

24. Server A
Server B
Server C
Server D
Server E
set
get
Adding nodes dynamically

25. get
set
Consistent Hashing for
Reading
Consistent Hashing for
Writing
Dynamic rebalacing algorithm of kumofs
（double-hash-space）

26. Application
Gateway
Gateway
Application
Server Server Server
memcached protocol
MessagePack-RPC
・Hides cluster configuration from
applications.
memcached server on localhost
localhost:11211
Asynchronous RPC library

27. MessagePack-RPC
Server Server Server
Tools
Administrator
Manager
Gets cluster configuration
Asynchronous RPC library

28. Tools
Written in Ruby
> automates operation tasks
> easy to implement
> easy to customize
kumoctl
kumostat
kumotop
MessagePack-RPC
Asynchronous RPC library
Cross-language RPC

29. Tools
Written in Ruby
> automates operation tasks
> easy to implement
> easy to customize
kumoctl
kumostat
kumotop
MessagePack-RPC
Asynchronous RPC library
Cross-language RPC
MessagePack

30. What s MessagePack?
• Binary-based object serialization format
• fast and small
• Cross-language
• Java, C++, Ruby, Python, Erlang, Haskell, ...
• Dynamic typing like JSON,Avro, etc. (Unlike Thrift)
• type-system is compatible with JSON
• Streaming deserialization

31. MessagePack?
{“msgpack”:“json”, “hello”:“world”}
24% compact
MessagePack
26 bytes
JSON
34 bytes
It's like JSON, but very fast and small.

32. It's like JSON, but very fast and small.
MessagePack?
Faster!
MessagePack for Java

33. require 'msgpack' # gem install “msgpack"
msg = [1,2,3].to_msgpack #=> "x93x01x02x03"
MessagePack.unpack(msg) #=> [1,2,3]
MessagePack?
It's like JSON, but very fast and small.

34. require 'msgpack' # gem install “msgpack"
u = MessagePack::Unpacker.new($stdin)
u.each do |obj|
puts obj.to_json
end
MessagePack?
It's like JSON, but very fast and small.

35. MessagePack?
Annotation + dynamic code generation in Java
@MessagePackMessage
public static class MyClass {
public String str;
public double num;
@MessagePackOptional
public int flag = 0;
}

36. MessagePack
• C++
> Sadayuki Furuhashi
• Ruby
> Sadayuki Furuhashi
• Java
> Muga Nishizawa
• Python
> INADA Naoki
• Haskell
> Hideyuki Tanaka
• Lua
> Nobuyuki Kubota
• Perl
> tokuhirom, gfx, ...
• Erlang
> UENISHI Kota
• Node.JS
> Peter Griess
• JavaScript
> uupaa
• PHP
> advect
• D
> repeatedly
Authors

37. MessagePack-RPC?
Rails app
C++ serverJava app
MessagePack-RPC
Cross-language messaging library

38. What s MessagePack-RPC?
• Cross-language messaging library
• Java, C++, Ruby, Python, Erlang, Haskell, ...
• Simple protocol
• 3 messages: Request, Response and Notify

39. What s MessagePack-RPC?
• Asynchronous RPC with parallel pipelining
• Concurrent communication over multiple servers
• Connection pooling
• omits overheads of reconnecting
• Concept of Future
• hides complicated event-driven I/O
• Concept of Session
• hides transport connections
• Dynamic typing

40. Protocol
MessagePack-RPC protocol
[0, msgid, method, params]
[1, msgid, error, result]
[2, method, params]
Request
Response
Notify
Message ID
→Parallel Pipelining

41. Pipelining
Client Server Pipelining
Sends one request after
another before receiving
responses to speed-up.
Examples:
・HTTP/1.1 pipelining
・Thrift
・SOAP

42. Client Server
Pipelining
heavy task
light task
Examples:
・HTTP/1.1 pipelining
・Thrift
・SOAP
Heavy task may keep
light tasks waiting
> makes parallelism worse
must be
sorted

43. Client Server
Parallel pipelining
Example:
・MessagePack-RPC
MessagePack-RPC doesn t
require responses to be sorted.
> Returns light tasks faster
Faster
heavy task
light task

44. Future
Client Server1 Server2
call_async(method1)
call_async(method2)
future1
future2

45. Event-driven I/O
Server-side event-driven I/O
Dispatcher
Server
Client
Client
Loop
multi-threaded event-driven I/O
(C++, Java)
Servers deals with many clients efficiently.

46. Event-driven I/O
Client
Session Loop Server
Client
ServerSession Loop
Client-side event-driven I/O

47. Event-driven I/O
shared
event loop
Client
Client
Server
Server
Loop
Session
Session
Client-side event-driven I/O
communications
concurrently

48. Event-driven I/O
Session Pool
Server
Server
pools these
connectionsLoop
Session
Session
connection
Client-side event-driven I/O

49. Dispatcher
Event-driven I/O
shared
event loop
Client
Server
Server
Loop
Session
Session
Client-side event-driven I/O
Client

50. shared
event looptimer, signal handler,
other protocol, etc...
Dispatcher
Event-driven I/O
Client
Server
Loop
Session
Client-side event-driven I/O
Client

52. MessagePack-RPC
• C++ - Sadayuki Furuhashi
• Erlang - UENISHI Kota
• Haskell - Hideyuki Tanaka
• Java - Muga Nishizawa
• PHP - h0x10
• Python - INADA Naoki
• Ruby - Sadayuki Furuhashi
Authors

53. Users
• Ameba Now
• Microblog service; 18 million posts per month
• Apache Solr + MessagePack for realtime search
• Sedue
• Commercial distributed search engine

54. http://sedue.sc/
Sedue Cloud Search: 簡単に使える高性能検索エンジン・レコメンドエンジンクラウドサービス

56. Conclusion
• Kumofs - Distributed key-value store
> Optimized for low latency
always zero-hop
> No SPOF
> Scales-out without any impacts on
applications
Gateway hides cluster configuration
Consistency control algorithm

57. Conclusion
• MessagePack
> Cross-language object serialization library
> Supports optional fields
• MessagePack-RPC
> Cross-language messaging library
> Server/client-side event-driven I/O
> Asynchronous RPC with parallel pipelining
> Concept of Future and Session
> Connection pooling

58. http://msgpack.org/
http://kumofs.sourceforge.net/